
United States R\tent and TkAPEMARK Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Addresj: COMMISSIONER OF PATENTS AND TRADEMARKS 
RO. Box 1450 

Alexandria, V^iginia 22313-1450 
www.vuprto.gov 



APPLICATION NO. 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


09/667,097 




09/21/2000 


Kumi Jinzenji 


10746/21 


6354 


26646 7590 


06/02/2003 









ONE BROADWAY 
NEW YORK, NY 10004 



EXAMINER 



GOOD JOHNSON, MOTILEWA 



ART UNIT 



PAPER NUMBER 



2672 

DATE MAILED: 06/02/2003 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 07-01) 



Office Action Summary 



Application No. 

09/667,097 



Examiner 

Motilewa A. Good-Johnson 



Applicant(s) 

JINZENJI ET AL 



Art Unit 

2672 



- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 . 1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )K Responsive to communication(s) filed on 04 March 2003 . 
2a)K This action is FINAL. 2b)n This action is non-final. 

3) 0 Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 0,G. 213. 
Disposition of Claims 

4) ^ Claim(s) 1-30 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) 0 Claim(s) is/are allowed. 

6) S Claim(s) 1-18.20 and 22-30 is/are rejected. 

7) K Claim(s) 17.19 and 21 is/are objected to. 

8) 0 Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) n The specification is objected to by the Examiner. 

10)0 The drawing(s) filed on is/are: a)n accepted or b)^ objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
1 1 )□ The proposed drawing correction filed on is: a)n approved b)n disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) 0 The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§ 119 and 120 

13) 0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)nAII b)n Some*c)n None of: 

1 Certified copies of the priority documents have been received. 

2.n Certified copies of the priority documents have been received in Application No. . 



3.n Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) 0 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) n The translation of the foreign language provisional application has been received. 

15) 0 Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
Attachment(s) 



1 ) O Notice of References Cited (PTO-892) 

2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) □ Infonnation Disclosure Statement(s) (PTO-1449) Paper No(s) . 



4) O Interview Summary (PTO-413) Paper No(s). 

5) □ Notice of Infonnal Patent Application (PTO-152) 

6) n Other: 



U.S. Patent and Trademark Office 
PTO-326 (Rev. 04-01) 
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DETAILED ACTION 

1 . This office action is responsive to the following communications: 
Application, filed on 09/21/2000; IDS, paper #2, filed on 09/21/2000; Request for 
reconsideration, filed 03/04/2003. 

This action is made final. 

2. Claims 1-30 are pending in this application. Claims 1, 3, 5, 7, 9, 10, 12, 
13, 15, 16, 18, 20, 22, 24, 25, 27, 28 and 30 are independent claims. No claims 
have yet been amended. 

3. The present title of this application is "Method for separating background 
sprite and foreground object and method for extracting segmentation mask and 
the apparatus" (as originally filed). 



Claim Rejections - 35 USC § 102 

4. The following is a quotation of the appropriate paragraphs of 35 
U.S.C. 102 that form the basis for the rejections under this section made in this 
Office action: 

A person shall be entitled to a patent unless - 

e) the invention was described in (1) an application for patent, published under section 122(b), 
by another filed in the United States before the invention by the applicant for patent or (2) a 
patent granted on an application for patent by another filed in the United States before the 
invention by the applicant for patent, except that an international application filed under the 
treaty defined in section 351(a) shall have the effects for purposes of this subsection of an 
application filed in the United States only if the international application designated the United 
states and was published under Article 21(2) of such treaty in the English language. 

5. Claims 1-15 are rejected under 35 U.S.C. 102(e) as being anticipated by 



Crinon et al., U.S. Patent Number 6,249,613, "Mosaic Generation and Sprite- 
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Based Coding with Automatic Foreground and Background Separation", class 
382/236. 



As per independent claim 1 , a foreground object and background sprite 
separation and extraction method for extracting . . . comprising the steps of: 
obtaining a global motion for transforming a coordinate system between a 
reference . . . ; mapping an original image corresponding to said frame into a 
reference coordinate system , . . ; generating a provisional sprite . . , ; cutting out 
a first image . . , using said global motion; obtaining a difference image between 
said first image and said original image; extracting a foreground object image . , . 
and extracting other region as a background image; mapping said background 
image . . . Crinon discloses segmenting foreground and background objects, 
coding in global motion parameters, col. 2, lines 37-55. Crinon further discloses 
reconstructing a sprite for isolating the object having the most motion in the video 
sequence, col. 3, lines 23-35. 

With respect to dependent claim 2, cutting out a second image from said 
background sprite . .. ; obtaining a difference image . . . ; extracting a foreground 
object image as a region in said difference . . . Crinon discloses in figure 8. 

As per independent claims 3 and 5, they are rejected based upon similar 
rational as above independent claim 1 respectively. 

With respect to dependent claims 4 and 6, they are rejected based upon 
similar rational as above dependent claim 2. 
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As per independent claim 7, a segmentation mask extraction method . . . 
comprising the steps of: receiving a foreground mask image . . . ; providing a 
first value as an alpha value ... in each of the first macro-blocks . . . ; providing 
said first value as said alpha value to all shape pixels in each of second macro- 
blocks . . . ; and outputting said segmentation mask. Crinon discloses first and 
second macro-blocks tagged as foreground and background objects, figure 9, 
col. 9, lines 42-67 and in col. 10, lines 1-30. 

With respect to dependent claim 8, receiving each of third macro-blocks . , 
. as said background part; and providing said first value to said third macro-block 
. . . Crinon discloses a binary segmentation map and further using a 
neighborhood of macro-blocks around a macro-block of interest, col. 9, lines 20- 
28. 

As per independent claim 9, a segmentation mask extraction . . . 
comprising the steps of: receiving a foreground mask image; generating a 
number map . . . ; initializing a foreground map; providing a predetermined value 
to each of positions in said foreground map . . . ; providing said predetermined 
value to each of position in said foreground map . . . ; and generating said 
segmentation mask . . . Crinon discloses generating macro-blocks having 
multiple local motion type vectors and further having macro-blocks as foreground 
only, background only, and foreground or background, col. 2, lines 37-67. 

As per independent claims 10 and 13, they are rejected based upon 
similar rational as above independent claim 7. 
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With respect to dependent claims 1 1 and 14, they are rejected based 
upon similar rational as above dependent claim 8. 

As per independent claim 12 and 15, they are rejected based upon similar 
rational as above independent claim 9, respectively. 



Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for 
all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described 
as set forth in section 102 of this title, if the differences between the subject matter sought to 
be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made to a person having ordinary skill in the art to which 
said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 



7. Claims 16, 18, 20, 22-30 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Wang, U.S. Patent Number 6,256,409, "Method for 
Determining a Correlation between Images using Multi-Element Image 
Descriptors", class 382/170. 



As per independent claim 16, a segmentation mask extraction method . . . 
comprising the steps of: obtaining said difference image by calculating an 
absolute difference . . . initializing an energy map for each macro-block of said 
difference image; calculating energy values for said each macro-block; obtaining 
an average of said energy values; calculating a foreground ratio . . . ; and 
generating said segmentation mask . . . Wang discloses an energy map, feature 
vector or image descriptor to describe multi-band images or the correlation 
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between a first image and a second image, col. 5, lines 12-59. However, it is 
noted that Wang fails to disclose representing an energy map as a macro-block 
for each different image. It would have been obvious to one of ordinary skill in 
the art at the time of the invention to include macro-blocks as image descriptors 
in the invention of Wang to further describe the correlation of the first and second 
images for performing image matching. 

As per independent claims 18 and 20, they are rejected based upon 
similar rational as above independent claim 16. 

As per independent claim 22, a segmentation mask extraction method for 
extracting a segmentation mask . . . comprising: a first step of regarding each of 
first macro-blocks as the foreground when an energy value of said first macro- 
block which is obtained , . , ; a second step of regarding each of second macro- 
blocks as the foreground . . . Wang discloses an energy map, feature vector or 
image descriptor to describe multi-band images or the correlation of a first image 
and a second image. However, it is noted that Wang fails to disclose 
representing an energy map as a macro-block for each different image. It would 
have been obvious to one of ordinary skill in the art at the time of the invention to 
include macro-blocks as the image descriptors disclosed in Wang to further 
describe the correlation of the first and second images for performing image 
matching. 

With respect to dependent claim 23, iterating said second step for 
predetermined times, Wang further discloses re-processing and re-examining 
the image descriptors, col. 21, lines 4-17. 




Application/Control Number: 09/667,097 Page 7 

Art Unit: 2672 

As per independent claim 24, Wang further discloses comparing the 
image descriptors and categorizing the different image descriptors and belonging 
to a different image category type, col. 17, lines 32-67. However, it is noted that 
Wang fails to disclose representing an energy map as a macro-block for each 
different image. It would have been obvious to one of ordinary skill in the art at 
the time of the invention to include macro-blocks as the image descriptors 
disclosed in Wang to describe the correlation of the first and second images for 
performing image matching. 

As per independent claims 25 and 28, they are rejected based upon 
similar rational as above independent claim 22. 

With respect to dependent claims 26 and 29, they are rejected based 
upon similar rational as above dependent claim 23, 

As per independent claims 27 and 30, they are rejected based upon 
similar rational as above independent claim 24 respectively. 

Allowable Subject Matter 

8. Claims 17, 19 and 21 are objected to as being dependent upon a rejected 
base claim, but would be allowable if rewritten in independent form including all 
of the limitations of the base claim and any intervening claims. 

9. The following is a statement of reasons for the indication of allowable 
subject matter: The prior art cited in its entirety fail to render obvious dividing an 
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energy value by an average to obtain the macro-blocks to represent the energy 
values in the foreground and background maps. 

Conclusion 

10. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of 
time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire 
THREE MONTHS from the mailing date of this action. In the event a first reply is 
filed within TWO MONTHS of the mailing date of this final action and the advisory 
action is not mailed until after the end of the THREE-MONTH shortened statutory 
period, then the shortened statutory period will expire on the date the advisory 
action is mailed, and any extension fee pursuant to 37 CFR 1 .136(a) will be 
calculated from the mailing date of the advisory action. In no event, however, will 
the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Response to Arguments 

1 1 . Applicant's arguments filed 03/04/2003 have been fully considered but 
they are not persuasive. 

Applicant argues that Crinon fails to disclose the use of a provisional 
sprite. Crinon discloses the use of multiple sprites, col. 2, lines 37-55, to create a 
background sprite. Deleting foreground objects generates applicant's provisional 
sprite as claimed. Crinon discloses segmenting foreground and background 
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objects. Applicant argues that Crinon fails to disclose cutting out a second 
image from the background sprite by using the global motion. Crinon discloses 
an event list to be associated with the background sprite. The event list uses 
token representative images of the foreground, thus not the actual foreground 
image, for the object position of the sprite. Crinon further discloses the using the 
vertex of distinctive features of foreground objects to allow a user to choose 
between coarse to finer shapes of the foreground object positions in the 
background sprite, col. 8, lines 1-12. Applicant argues that Crinon fails to 
disclose cutting out a second image from the background sprite using global 
motion, based upon a difference image higher than a threshold. Crinon discloses 
scene changes or motion compensated areas to identify regions with changes in 
content to determine the background object and foreground object, col. 5, lines 
19-48, and performing a threshold to disclose the foreground from the 
background, col. 6, lines 1-41. Applicant further argues that Crinon fails to 
disclose a first and second macro-block equal to or larger than a predetermined 
value in the foreground map. Crinon discloses sub-classification of the regions 
tagged as 1, the foreground region, into la and 1b, for macro block coding, col. 
6, lines 26-37. Crinon further discloses the tagged regions are coded for different 
image support. 

Applicant argues that Wang fails to disclose generating of a segmentation 
mask using foreground ratio. Wang discloses an energy matrix referred to as an 
orientation map, calculating the dominant orientation at each pixel and 
determining a set of co-occurrence descriptors, such as maximum probability, 
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entropy, unifornnity, mean (ratio), etc., and combining the descriptors to form a 
feature vector or image descriptor, col. 5, lines 12-23. Wang further discloses 
determining a correlation between images, col. 2, lines 10-13, Crinon discloses 
mosaic images built from images of a scene and image segmentation, which is a 
correlation between images, thus providing the motivation to combine Wang with 
the invention of Crinon. Applicant further argues that Wang does not describe 
first and second macro-bloc approximations. Crinon discloses sub-classification 
of the regions tagged as 1 , the foreground region, into 1a and 1 b, for macro block 
coding, col. 6, lines 26-37. Crinon further discloses the tagged regions are coded 
for different image support. Wang discloses determining a correlation between 
images using an energy map. It is obvious that if Crinon allows for the sub- 
classification of the macro-blocks in the foreground region, the invention of Wang 
would allow for the coding of the sub-classification of the macro-blocks using 
energy map descriptors. 

Any inquiry concerning this communication or earlier communications from 
the examiner should be directed to Motilewa A. Good-Johnson whose telephone 
number is (703) 305-3939. The examiner can normally be reached on Monday - 
Friday 8:30 AM - 5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Mike Razavi can be reached on (703) 305-4713. The fax 
phone numbers for the organization where this application or proceeding is 
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assigned are (703) 872-9314 for regular communications and (703) 872-9314 for 
After Final communications. 

Any inquiry of a general nature or relating to the status of this application 
or proceeding should be directed to the receptionist whose telephone number is 
(703) 306-0377. 



Motilewa A. Good-Johnson 

Examiner 

Art Unit 2672 
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May 19, 2003 
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SUPERVISORY PATENT EXAMINER 
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